token() -- split a string into multiple strings

dim w$(10) 
... 
num=token(a$,w$()) 
num=token(a$,w$(),s$) 

The token-function accepts a string (containing the text to be split), a reference to a string-array (which will receive the resulting strings, i.e. the tokens) and an optional string (with a set of characters, at which to split, i.e. the delimiters).

The token-function regards its first argument as a list of tokens separated by delimiters and it will store the list of tokens within the array-reference that has been supplied. Note, that the array, which is passed as a reference (w$() in the synopsis), will be resized accordingly, so that you don't have to figure out the number of tokens in advance. The element at position zero (i.e. w$(0)) will not be used.

Normally (i.e. if you omit the third, the delimiter-argument) the function will regard space or tab as delimiters for tokens; however by supplying a third argument, you may split at any single of the characters within this string. E.g. if you supply ":;" as the third argument, then colon (:) or semicolon (;) will delimit tokens.
Note, that token will never produce empty tokens, even if two or more separators follow in sequence. Refer to the closely related split-function, if you do not like this behaviour. In some way, the token-function focuses on the tokens and not on the separators (other than the split-function, which focuses on the separators).

The second argument is a reference on a string-array, where the tokens will be stored; this array will be expanded (or shrinked) as necessary to have room for all tokens.
The first argument finally contains the text, that will be split into tokens. The token-function returns the number of tokens, that have been found.

Please see the examples below for some hints on the exact behaviour of the token-function and how it differs from the split-function:

print "This program will help you to understand, how the" 
print "token()-function exactly works and how it behaves" 
print "in certain special cases." 
print 
print "Please enter a line containing tokens separated" 
print "by either '=' or '-'" 
dim t$(10) 
do 
	print 
	input "Please enter a line: " l$ 
	num=token(l$,t$(),"=-") 
	print num," Tokens: "
	for a=1 to num 
		if (t$(a)="") then 
			print "(EMPTY)" 
		else 
			print t$(a); 
		endif 
		if (a<num) print "," 
	next a 
	print 
loop 

This program prints the following output: 
  

Please enter a line: a 
1 Tokens: a 

Please enter a line: 
0 Tokens: 

Please enter a line: ab 
1 Tokens: ab 

Please enter a line: a=b 
2 Tokens: a,b 

Please enter a line: a- 
1 Tokens: a 

Please enter a line: a-= 
1 Tokens: a 

Please enter a line: =a- 
1 Tokens: a 

Please enter a line: a=-b 
2 Tokens: a,b 

Please enter a line: a--b- 
2 Tokens: a,b 

Please enter a line: -a==b-c== 
3 Tokens: a,b,c